Regression Adjustments to Experimental Data: Do David Freedman’s Concerns Apply to Political Science?” Paper presented at the annual meeting of the Society for Political Methodology

Abstract: One of David Freedman’s important legacies was to raise awareness of the assumptions that underlie everyday statistical practice, such as regression analysis. His recent papers (Freedman 2008a, 2008b) offer stern warnings to those who offer regression analysis as an appropriate way to analyze experimental results. In particular, Freedman demonstrates that including pre-treatment covariates as controls leads to bias in finite samples and inaccurate standard errors. Freedman advises researchers against using regression adjustments for experiments involving fewer than 500 observations (2008a, p.191), a recommendation that has gained increasing attention and acceptance among social scientists. This paper argues that the ever-cautious Freedman was probably too cautious in his recommendations. After explicating the special features of Freedman’s model, I use a combination of simulated and actual examples to show that as a practical matter the biases that Freedman pointed out tend to be negligible for N> 20. Pathological cases that could generate biases for larger experiments involve extreme outliers that would be readily detected through visual inspection